Skip to content

volume: reset pool_id on failed volume create for zone-wide storage#13611

Open
nikolauseppinger wants to merge 1 commit into
apache:4.22from
nikolauseppinger:fix/zonewide-poolid-reset
Open

volume: reset pool_id on failed volume create for zone-wide storage#13611
nikolauseppinger wants to merge 1 commit into
apache:4.22from
nikolauseppinger:fix/zonewide-poolid-reset

Conversation

@nikolauseppinger

@nikolauseppinger nikolauseppinger commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Description

When createVolumeAsync fails, VolumeServiceImpl.createVolumeCallback resets the volume's pool_id only if volume.getPodId() != null. Zone-wide primary storage pools have no pod (pod_id is NULL), so on a failed create the volume is reverted to Allocated but keeps a stale pool_id.

On the next create/attach, VolumeOrchestrator.findStoragePool then returns no pool: StorageManagerImpl.storagePoolCompatibleWithVolumePool rejects the volume because its state is not Ready (it has a pool_id but is Allocated), so every candidate pool is filtered out. The operation fails with:

Unable to find suitable primary storage when creating volume [...]

even though the zone-wide pool is Up and has plenty of free capacity. The only recovery is a manual DB fix (UPDATE volumes SET pool_id=NULL ...).

Root cause: the guard added in #10757 keys on getPodId(), which is only set for pod/cluster-scoped placement. The field actually being reset is pool_id, so the guard must key on getPoolId().

Fix: guard the reset on getPoolId() != null instead of getPodId() != null, in both createVolumeCallback and destroyAndReallocateManagedVolume. ensureVolumeIsExpungeReady is intentionally left unchanged (it legitimately clears pod_id).

Reproducible with encrypted volumes on a zone-wide SharedMountPoint pool: the first attach create fails (unrelated agent reason), leaving the volume stuck, after which all retries fail at findStoragePool.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

On CloudStack 4.22.1.0 with a single zone-wide SharedMountPoint primary storage (KVM):

  1. Trigger a failed volume create on the zone-wide pool (e.g. an encrypted DATADISK whose disk creation fails on the agent).
  2. Before: volume stays Allocated with pool_id still set; every subsequent attach fails with Unable to find suitable primary storage.
  3. After (patched management server): on the failed create the pool_id is reset to NULL, the volume returns to a clean Allocated state, and the next attach picks the pool normally (verified: no manual DB intervention needed).

When createVolumeAsync fails, createVolumeCallback resets the volume's
pool_id only when volume.getPodId() != null. Zone-wide primary storage
pools have no pod, so such a volume keeps a stale pool_id while it is
reverted to Allocated. On the next create/attach, findStoragePool then
returns no suitable pool because storagePoolCompatibleWithVolumePool
rejects the volume (its state is not Ready), and the operation fails with
"Unable to find suitable primary storage" even though the pool has plenty
of capacity.

Guard the reset on the field that is actually being cleared
(getPoolId() != null) instead of getPodId(), so it also applies to
zone-wide (and local) storage. The same guard is fixed in
destroyAndReallocateManagedVolume. ensureVolumeIsExpungeReady is left
unchanged as it legitimately clears pod_id.

Regression from apache#10757.

@weizhouapache weizhouapache left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants